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ABSTRACT 

The purpose of the audit report on the Southeastern 
New Mexico Bilingual Program (1972-73) of the Artesia School District 
is to give a critique of the project's evaluation with notations on 
comparative findings of the project evaluation and the audit, and to 
confirm or question the program modifications proposed in the 
evaluation. The 5 audit report sections are: (1) introductory and 
general comments on the project evaluation quality and comparative 
findings of the project evaluation and the audit; (2) detailed 
critique of the comprehensive evaluation for each conponent, hased on 
an assessment of the instruments used, data collection, and data 
analysis techniques, and presentation; (3) description of the 
auditor's on-site findings and correlation with the evaluator's data 
and reports on a component by component basis, and a summary of 
consistencies and discrepancies; (4) general recommendations for 
ev2U.uation design revisions with a rationale for each; and (5) 
confirmation or questioning of the need for program modifications 
proposed in the project evaluation. (NQ) 
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EDUCATIONAL ACCOMPLISHMENT AUDIT REPORT 



In compliance with the contractual agreement, legal 
prescriptions , and official directions under the provis ions 
of Title VII of Public I^w 89-10, as amended, for the estab- 
lishment of bilingual education programs the following educa- 
tional accomplishment audit report is being submitted on the 
Southeastern New Mexico Bilingual Program (OEG-0-9-41002&'' 
3443- (280). The program is funded through the Artesia (New 
Mexico) School District with Bill Gutierrez as Project 
Director . 

The audit report as submitted herein, is a critique 
concerning the project evaluation with notations on compa- 
rative findings of the project evaluation and the audit is 
related to the performance and process objectives established 
in the evaluation design instrument. The auditor, in this 
report, will confirm, question, or suggest the need for pro- 
gram modifications which have been proposed as a result of 
the project evaluation and onsite visitations and observations. 

Scope of the Audit Report 

The audit report follows rather closely the Suggested 
Audit Report Content Areas received from the United States 
Office of Education and made a part of the contractual agree- 
ment between the Educational Accomplishment Auditor and the 
Project Director, as follows: 

1. Introductory and z,eneral comments concerning the 
quality of the project evaluation and the comparative 
findings of the project evaluation and the audit. 

2. Detailed critique of the comprehensive evaluation 
conducted, by component, based on an assessment of the 
instruments inside, data collection, data analysis, and 
the manner in which data analysis is presented* 

3. Description of the audit contractor's onsite 
visit findings and their correlation with the project 
evaluator's data and reports on a component by component 
basis; summary of consistencies and discrepancies, and 
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interpretation of the discrepanci^^s . 

4. Reconunendat ions for revisions in the evaluation 
design, including a rationale for each recommendation. 
Since the auditor's objectivity can be maintained only 
if the selection of a specific corrective action is a 
local decision, the audit contractor will provide gene^^al 
rather than specific recommendations, posing se\'^ral 
alternative actions on possible sources of assistauct ic 
the project director in correcting the deficiency. 

5. Confirmation or questioning of the need for 
program modifications which have been proposed as a 
result of project evaluation. 

Each of the audit areas d«;:scribed above will be 

referred to as sections with section number agreeing with .he 

audit description number. 

SECTION I -INTRODUCTORY AND GENERAL COMMENTS 

The Southeastern New Mexico Bilingual Program, 
OEG-0-9-410028-3443-(280), is sponsored by the Artesia (New 
Mexico) Public Schools and is under the direction of a 
Project Director who along with other available specialists 
from within the project and/or school system provides 
managerial and supervisory leadership. 

Six public school districts (Artesia, Carlsbad, 
Dexter, Hagerman, Lake Arthur, and Roswell) with supplemen- 
tary and complementary help from Eastern New Mexico University 
Roswell Campus and the New Mexico State Department of Educa- 
tion have cooperated to implement, and continue, the Bi- 
lingual Program in. the Lower Pecos Valley of New Mexico. 
The cooperating schools are participating in the planned 
program on a daily basis for a period of nine (9) months. 
Artfsia is in the fourth year phase of the project while the 
other schools are involved in their third year. 

All classes participating in the project are housed 
in regular classrooms within che confines of an elementary 
schcoJ plant* Each classroom component operates within the 
regular hours of the respective elementary school. School 
policy relating Xo personnel, students, and general building 
managemer^t and operation extends to all personnel and 
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students in the bilingual and laboratory components. In all 
components where a monolingual laboratory teacher is involved, 
bilingual sessions are provided within the morning schedule 
since the teacher aide assigned to each classroom being 
serviced works in the classroom only in the morning. The 
paraprofessionals in the project attend the college program 
•during afternoons and evenings. 

The effectiveness of an evaluation is primarily 
determined by how well the evaluating techniques quantify 
through objective measurement and/or through individualized 
student needs assessment, the quality and extent of pupil 
behav^ioral development for which the project is chiefly 
rospor.sib le . After careful examination of the evaluation 
design, techniques employed in the evaluation, procedures 
follow, and purusal of the written evaluation report it can 
be concluded that the major objectives have been to measure 
learning outcomes, progress toward accomplishment of program 
objectives, and a realistic assessment of individual 
accomplishments and needs. The evaluation, although cogni- 
zant of the inputs, has concentrated its evaluative judg- 
ments on outputs - progress toward achievement of stated 
project objectives. 

Standardized instruments, uniformly applied to 
experimental (project participants) groups and control 
groups have been utilized for objective measurements. The 
instrument used for cognitive evaluation of general academic 
accomplishment for grades three and four was the Compre- 
hensive Tests of Basic Skills published by California Test 
Bureau. The C.T.B.S. was selected because it follows the 
statewide pattern of student assessment and enjoys generally 
high acceptance as a standardized measuring device with high 
validity in normal applications. Analysis of test results 
have been divided into four groups : 

1. Dominant Spanish speaking children in bilingual 
components . 

2. Non-Dominant Spanish speaking children in the 
bilingual component. 



3. Dominant Spanish SDeaking children in the 
laboratory component. 

4. Non-Dominant Spanish speaking children in the 
laboratory component. 

Crirerion Reference Skills Charts were maintained 
by teachers in grades one and two. These cliarts were de^/el- 
oped for each child and covered areas of reading, mathem^atics , 
social studies, science, English as a second language, 
Spanish as a second language, and multicultural development. 
Analysis of these individual records has established the 
areas of accomplishment and identify areas of student need. 
Validity of the Skill Charts is dependent upon the profes- 
sional judgment of the teacher as she examines the accomplish- 
ments of each indiv^ldual student in each skill activity in 
the various Skill Charts. 

Informal assessments based on observations, teacher 
made measuring instruments and student performance were 
utilized to provide feedback during the teaching- learning 
process as supplementary instruments (or processes) for 
project evaluation purposes. 

The evaluation design, procedures, and techniques 
were developed and refined during and following a series of 
conferences between the project director, the evaluator, and 
the auditors. Communication channels have remained open and 
have served to retain excellent rapport. Through the dedi- 
cated and cooperative efforts of all concerned it has been 
possible to eliminate irrelevant procedures and the refine- 
ment of evaluation procedures to better assess the level, or 
degree, of accomplishment of program objectives and at the 
same time providing for identification of specific needs of 
individuals , 

It is unfortunate that the project has found it 
necessary to contract with a different evaluator for each 
year it has been in operation. Circumstances have made this 
unavoidable, and in fact, in some ways perhaps there has been 
some benefits in this, in that the ev^aluation to date re- 
flect the analyses and interpretation of four separate but 
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qualified persons. The current evaluator has approached the 
evaluation process in a serious manner, has brought a 
wealth of professional insight into the evaluation, and has 
functioned with a high level of professionalism. Her report 
reflects all of these qualities. 



SECTION II - CRITIQUE OF EVALUATJON REPORT 



The Evaluation Design 

The Evaluation Design was developed to measure the 
extent to which perfortiiance objectives wore attained. The 
design is well organized and meets the basic needs for 
effective program evaluation. 

The Evaluation Instruments 

The evaluation instruments consist of the following: 

1. The California Test of Basic Skills was designated 
as the basic instrument to measure the instructional 

and English language arts componencs. It was selected 
largely because it is part of the state testing program. 
It is a well recognized test and has high validity. 

2. The Self-Image Test, a project devised xest that 
is highly subjective, does function to give some indi- 
cations of the Ftxident's se If-irriage . 

3. The Spanish Language Arts Criterion Reference 
Test, developed and used locally for the first time, 
appears adequate to determine baseline data and achieve- 
ment related specifically to the program content and 
objectives . 

4. The Otis-l*ennon Mental Ability Test was given 
city-wide to ^ first -grade students in the fall, and was 
therefore a useful instrument to assess the readiness and 
intellectual level of first graders in the bilingual 
program. 

5. The Metropolitan Achievement Test was given to 
all city students in the second grade. The scores from 
this test were used to help determine the maturity and 
ability levels of the second grade children. 

Control Groups . The Control Group was maintained for 
third and foi^rth grades because of high mol^ility, some problems 
were encountered in retaining a matched grouping large enough 
to yield significant comparative data. Through a system of 
^^rematching" other students were iiicluded in the control group. 

ERLC 
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Audit of Forf ormance Objectives 
Ac c u i ip 1 is f ii r.c n L s 

Stan d ard curriculum performance . In examination of 

the evaluation report and inspection of test records and 

data IL is possible to ascertain that the project objective 

relative to growth in standard curriculum (reading, arithmetic 

and language) for grades three and four dominant Spanish 

speaking have been properly analyzed by the project evalu- 

ator. The results show that the fourth grade bilingual group 

far exceeded the control group in reading and language. The 

accomplishment in arithmetic, although near average foi' each 

group, shows very little difference in amount of per pupil 

gain. 

For the third grade, the bilingual group showed 
marked gain over the control group in all three areas. The 
assessment as made by the evaluator is accurate, and it can 
be concludou that the objective was not only met but actual 
growth far exceeds the stated objective. 

The non- Spanish dominant language third and fourth 
graders present an interesting picture on standard curric- 
ulum performance. The fourth grade bilingual group out- 
performed the control group in all areas (reading, arithmetic, 
and language) to a degree well above that stated in the 
objective. However, at the third grade level the control 
group demonstrated performance on all three tests signifi* 
cantly above that of the bilingual group. 

The evaluator *s assessment that the fourth grade 
surpassed th^ project objective while the third grade failed 
to reach the objective is an accurate assessment. 

The data for first and second grades was analyzed 
and the statistical treatment indicates that positive gains 
were made by students in the standard curriculum. 

Instructional and English language component . In 
response to an informal questionnaire ceacher assessment of 
student performance shows that teachers generally felt that 
students were achieving at least five percent (5%) more of 
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the standard curr icuLv.n. Response to items on the question- 
naire ranged from small gain assessment to large gain, with 
teachers generally feeling that children in the bilingual 
program were making positive performance development • The 
evaluators interpretation of teacher assessment is accurate. 
From the results of the questionnaire it can be concluded 
that, in the opinion of teachers in the program, bilingual 
instruction is no hindrance to the achievement in the English 
Language component , 

Spanish language component . In analysis of the ob- 
jective relative to facility in use of Spatiish langiiage the 
following assessment summary is made by the evaluator: 

Grade four, dominant Spanish and non-dominant Spanish 
speaking bilingual groups both exceeded the performance 
of the corresponding control groups well above the level 
projected in the stated objective. 

Grade three, dominant Spanish and non-dominant Spanish 
speaking bilingual groups both exceeded the performance 
of the corresponding control groups well above the level 
projected in the stated objective. 

In grade two the non-dominant Spanish speaking bilin- 
gual group failed to attain the seventy percent (70^*0 
level on th^ Spanish Language Arts Criterion Reference 
Test by a slight 1.4 points. The first grade and second 
grade dominant Cpanish 5? peaking group and the first grade 
non-dominant jpanish speaking groups attained or exceeded 
the sev^enty percent (70%) criterion established by the 
objective . 

Examination of tests and test results by the project 
auditor confirms this assessment. 

Self-ima;:;e component . The evaluators analysis and 
assessment of the objective that bilingual program students 
will have a five percent (5%) better self-Image than control 
students indicates that grades two and four significantly 
exceeded the objective while grades one and two were slightly 
below the projected greater growth in self*image. The eval- 
uator 's notation that while the tself-image score results 
for bilingual program children was positive at all grade 
levels, while losses were registered for third and fourth 
grade control groups suggests that the bilingual program is 

ERLC 



effective in helping these children hold a relatively good 
self-image. The auditors acknowledge this observation as 
signif leant . 

When teachers were asked to respond to the part of 
the questionnaire related to the students' self-Image, their 
responses support the analyses of the self-image rating 
scale responses by students. 

English- Spani s h teaching; :.omponent . The process 
objective that "all children will be taught by bllinpual 
teachers utilizing both English and Spanish for teaching 
purposes on a fifty percent (50%) classroom time basis" was 
analyzed by the evalixator as summarized rn the following 
manner: (a) it was not possible to verify whether equal 
classroom Instructional time was equnlly c'.vided between 
English and Spanish, (b) teachers and aides generally felt 
that their students did receive adequate preprration to 
operate bilingually, (c) observation led to tne conclusion 
that a disparity did exist among classrooms in the use of 
Spanish in the instructional com;>onent , and (d) although 
aides were being given teaching responsibility, to expect 
twelve and one-half hours per week of instruction utilizing 
Spanish requires aides to tetch constantly in Spanish since 
they are in the classroom on a par*:*tinie basis. 

SECTION III - AUDITOR »S ON-SITE VISITATION 

During the course of the 1^72-73 school year the 
auditing team spent a total of fourteen man-days in on-site 
visitation. Interim reports were submitted earlier. The 
following is a report of visitations near the end of the 
teaching year. 

The specific purposes of the on-site visitation were 
(a) to assess the role and input of Artesia Public School 
Administration in the Title MI Program, (b) to determine 
Whether the teacher aide training program objectives were 
being met, (c) to observe whether teachers were making 
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effective use of their aides, and (d) to study raw data and 
other pertinent information related Lo the evaluation of the 
program. 

Administration of the Program 

The project director and curriculum specialist work 
effectively and have given a realistic and educationa ly 
sound meanings understood by all participants, to the phi- 
losophy and function of bilingucl education and to the con- 
tent of the curriculum. 

On-site visits have served to establish that the 
principals are cooperative and understand the programs 
operation. The project director is consulted by principals 
in matters relating to personnel, parental involvement, and 
pupil participation involved in all facets of the project. 
It has been established that teachers and aides are enthu- 
siastic about bilingual program activities and are well in- 
formed regarding the curriculum. The school community has 
come to endorse, accept, and understand the bilingual program 
as it is currently being administered by the Artesia Public 
Schools. Upward expansion of the program is following a 
"continued growth" pattern. 

The personnel continue to demonstrate a willingness 
to work in modifying and adaptx^^ the curriculum content to 
meet the individua 1 educat iona 1 needs of children in the 
various classroom units of the project. 

School administrators (superintendent , principals , 
curriculum coordinator, etc.) have cooperated in developing 
working relationships with all those assigned to the bi- 
lingual program as a means of enhancing an effective educa- 
tional experience for children. 

Teacher-aide training program . The Grant Package 
Proposal for FY 1972-73 included an ambitious, but realistic, 
program for the para-professional training component of the 
project. The agreement was continued with Eastern New 
Mexico University-Roswell Campus to continue the training 
O of the oara-prof essionals . At the present time ENMU-Roswell 

ERIC 
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Campus offers para-professionals a program of studies which 
enable the participants to eventually acquire a Bachelor of 
Arts degree in elementary education with a major in the 
teaching of ethnic groups and a minor in Spanish. Some 
thirty participants are taking advantage of this opportunity. 
Cultural Anthropology, Ethnology of the Southwest, Selected 
Topics in Spanish (a beginning field course in Bilingual 
Instruction), Hispanic Culture and Civilization, and all 
levels of Spanish courses are but a few of the prescribed 
offerings. It is the feeling of the auditing team that the 
pre-service and in-service training programs as implemented 
have provided the participants with a vast and useful body 
of knowledge regarding modern theories of human growth and 
development and a realistic philosophy of education. We 
agree with the Project Director when he states that "the 
Leacher-aide training component of the project has offered 
stability and scope toward the gradual training of bilingual 
teachers to fill the vast need of Southeastern New Mexico." 
Another significant strength of the prescribed training 
component (pre^ervlce and in-service) Is that although 
methods for implementation of the curriculum are a basic 
part of the agenda, it is child-oriented wben one studies the 
content and guidelines offered to the participants. Parents, 
teachers, teacher-aides and pupils are involved in the 
planning. The input provided in this manner is valued by 
both the Project Director and the Curriculum Specialist. 
Incidentally, the latter two professionals go about their 
business with dedication and not with assumed superiority 
as two "generals" of c\ militaristic extravaganza. Their 
sincere dedication serves as a model fo^ other participants 
in the program. 

Teacher use of teacher -a Ide s . This component of the 
program has had more ei;\phasis during FY 1972-73. It is a 
major component of any effective bilingual-bicultural program. 
The teacher-aide is functioniiig as a second voice for the 
teacher by assisting with tae Spanish instructional 
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reinforcement; the implications of impornance are obvious. 

At several of the centers visited teachers were more 
willing to allow the teacher-aide to assume the instruction 
of the related curriculum in Spanish. It has been noted 
that all participating teachers were regarding their aides 
as part of the teaching team and that planning for the day's 
activities is indeed a cooperative enterprise involving 
both. Significant improvements were noticed in follow-up 
visits . 

The evaluation . Cooperative efforts and sincere 
interest in the program by those associated with the pro- 
gram made it possible to meet all obstacles with satisfactory 
alternative solutions. The evaluation report as suti;.itted 
has been thoroughly critiqued elsewhere. Suffice ir here to 
merely comment that the evaluation was done in a very pro- 
fessional manner and presented with as much ohject\vity as 
was possible. There was no evidence of any actem^c to white- 
wash any phase of the program. On several occasions expla- 
nations and such conclusions were arrived at or. ly after very 
thoughtful consideration. 

Conclusion . The auditing experience, particularly 
the on-site visitation, has been a most pleasant one. The 
courtesies extended by all of the Bilingual Program staff and 
school acminis trators are indeed appreciated and most grati- 
fying. The auditing team sincerely submits that the South- 
eastern Bilingual Program might well serve as a model for 
this region, as for four years it has dated to be creative, 
innovative, flexible, and unafraid of publishing the factual 
results of its effort using as a base its belief that "this 
program, is a vehicle and pedigogical tool to better prepare 
children to fi:nction as successful citizens in a democratic 
society." It i^; firmlv^ believed that this program is bring- 
ing to ftrtesia and er.virons interracial unity and under- 
standing. 
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SECTION - REVISION IN EVALUATION DESIGN 



The Evaluation Design as utilized during the 1972-73 
school year has proved to be effective and stimulator for 
more effective instruction. It is the auditor's recom- 
mendation that the 1972-73 Evaluation Design instrument be 
utilized for 1973-74 with the following possible modification: 

1. Consideration should be given to either revising 
the self-image test or the acquisition of another in- 
strument for this purpose. Reference was made to this 
in the auditor ^s interim report and the evaluator's 
final evaluation reports 

2. A plan for pre-tcsting on the criterion referenced 
tests should be developed and implemented at the begin- 
ning of the 1973-74 school year. 

3. The criterion cards used for maintaining indivi- 
dual student progress records should be periodically 
reviewed by the project director and the evaluator 
throughout the school year. 



SECTION V - NEEDS FOR PROGR.AM MODIFICATION 



The Southeastern New Mexico Bilingual Program has 
just completed its fourth successful year of operation. The 
program in the Artesia Public Schools has advanced through 
the four year (grade) level of school. 

The successes and weaknesses of the project have 
been identified through visitations, observations, informal 
assessments, and the formal evaluation of the project. 

On the basis of t^iese opportunities for assessing 
the program the auditors present the following suggestions: 

1. Establish more clearly the differentiation be- 
tween the bilingual classrooms and the laboratory class- 
rooms in order tha.^: the strengths and weaknesses of each 
might be more clearly identified. This is also suggested 
as a means of helping school personnel not identified 
with the program anc interested townspeople better under- 
stand the program. 

2. Establish some framework for analysis of cost- 
quality re latioTJShips of both the bilingual and the 
laboratory components. These should be developed in 
such manner that the dollar yalwe of services received 
can be compared with the pupil cost in the "regular" 
classroom. 



3. Earlier assessments have shown that students' 
self-image suffers significantly at the intermediate 
grade levels . Serious consideration for extending the 
bilingual program into the fifth year (grade) with 
special emphasis being placed on the affective behavioral 
domain should be a priority item for the Artesia Public 
Schools . 

4. Elementary building principals need to become 
more actively involved. Ultimately the program will 
become their responsibility as part of the elementary 
school program. Every effort should be made to involve 
them in all inservice activities. Likewise, principlas 
should actively engage in future project planning. 

5. Efforts should continue to encourage teachers to 
plan for, and utilize, teacher aide services more 
extensively as bilingual adjuncts in the classroom. 

6 . Time ut i lizat ion is important in order that both 
teacher and aide function at peak efficiency. Careful 
planning should become a part of role expectations for 
all personnel in the program in order that the best 
possible utilization of available time and professional 
talents will be assured. 



CONCLUSIONS 



The Southeastern New Mexico Bilingual Program has 
successfully completed its fourth year of operation. During 
this time the program has expanded upward and into other 
cooperating school districts. 

Although staffing primary grades with well qualified 
bilingual teachers is becoming increasingly difficult , the 
Southeastern New Mexico Bilingual Program has successfully 
met this obstacle and the program has moved forward with no 
significant interruptions . 

Mr. Bill Gutierrez, Project Director, has worked 
unceasingly toward the accomplishment of project goals. He 
has exerted positive leadership in involving staff members 
in planning and program development. Mr. Gutierrez enjoys 
high respect from his staff, the administration, and the 
downtown community. 

The staff has proven itseif to be dedicated and 
anxious to develop the best possible bilingual program to 
serve the youth throughout the Lower pecos Valley. The 
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members of the staff have supported such project activities 
as inservice tra ining, program deve lopmenu , community in- 
volvement, and materials development. 

Although project evaluation presented some problems 
at times, it has been successfully completed and presented 
in a report that is readily understood. The report is 
accurate, frank, and to the point. Some areas failed to 
measure as strongly in the evaluation process as aimed for 
in the objective. Yet, generally speaking the project has 
demonstrated success. 

The project auditors have been well received by 
project staff members and administrators of all schools 
cooperating in the project. Records, reports, and materials 
were always available when needed. 

The Southeastern New Mexico Bilingual Project and 
its staff are to be commended for providing a forward thrust 
progr^.m which is contributing to the educational break- 
through for effective education for the dominant Spanish 
speaking child in the southwest. 



Respectfully submitted: 



'I * V. Payne , Chief Auditor 



Edward Medina, Auditor 
July 5, 1973 



